Method and apparatus to facilitate processing a stereoscopic image using first and second images to facilitate computing a depth/disparity image

ABSTRACT

The processing of a stereoscopic image using first and second images to facilitate computing a corresponding depth/disparity image can be facilitated by providing ( 101 ) the first and second images and then computing ( 103 ) a disparity value of each pixel in the second image by, at least in part, determining a likelihood of occlusion for at least some pixels comprising the second image using, at least in part, predicted occlusion information as corresponds to information contained in the first image. By one approach, this predicted occlusion information can be provided, at least in part, by processing ( 102 ) the first image, at least in part, by determining occlusion value information for at least some pixels as comprise the first image and then using the occlusion value information for the first image to determine a corresponding disparity information map for the first image.

TECHNICAL FIELD

This invention relates generally to the processing of stereoscopicimages.

BACKGROUND

Stereoscopic images are known in the art. Such images typically relateto using a first and a second image wherein both such images represent aview, from slightly different locations, of a same scene. Many times(and perhaps in keeping with normal human vision) the first image willcomprise a so-called left image and the second image will comprise aso-called right image.

In stereoscopic images, part of the scene as is visible in one of theimages will not be visible in the remaining image due to occlusion. Withinformation about such an occluded area being available from only oneimage, it becomes difficult to compute the depth of the occluded regionusing traditional triangulation calculations. Such occluded areas canalso make it more difficult to calculate depth for regions that neighborthe occluded areas.

Interest in stereoscopic images continues to grow. This includes, forexample, proposals to employ stereoscopic images in active displayplatforms such as, but not limited to, personally transportable devices(such as cellular telephones, personal digital assistants, and soforth). The above-noted problems with occluded areas, however, tend toreduce user satisfaction and/or to significantly increase the cost ofthe corresponding platform in order to better accommodate suchoperational circumstances. Existing approaches in this regard, forexample, tend to converge to a solution in a relatively slow mannerand/or via use of a relatively large store of available memory.

BRIEF DESCRIPTION OF THE DRAWINGS

The above needs are at least partially met through provision of themethod and apparatus to facilitate processing a stereoscopic image usingfirst and second images to facilitate computing a depth/disparity imagedescribed in the following detailed description, particularly whenstudied in conjunction with the drawings, wherein:

FIG. 1 comprises a flow diagram as configured in accordance with variousembodiments of the invention;

FIG. 2 comprises a block diagram as configured in accordance withvarious embodiments of the invention; and

FIG. 3 comprises a flow diagram as configured in accordance with variousembodiments of the invention.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions and/or relative positioningof some of the elements in the figures may be exaggerated relative toother elements to help to improve understanding of various embodimentsof the present invention. Also, common but well-understood elements thatare useful or necessary in a commercially feasible embodiment are oftennot depicted in order to facilitate a less obstructed view of thesevarious embodiments of the present invention. It will further beappreciated that certain actions and/or steps may be described ordepicted in a particular order of occurrence while those skilled in theart will understand that such specificity with respect to sequence isnot actually required. It will also be understood that the terms andexpressions used herein have the ordinary meaning as is accorded to suchterms and expressions with respect to their corresponding respectiveareas of inquiry and study except where specific meanings have otherwisebeen set forth herein.

DETAILED DESCRIPTION

Generally speaking, pursuant to these various embodiments, theprocessing of a stereoscopic image using first and second images tofacilitate computing a corresponding depth/disparity image can befacilitated by providing the first and second images and then computinga disparity value of each pixel in the second image by, at least inpart, determining a likelihood of occlusion for at least some pixelscomprising the second image using, at least in part, predicted occlusioninformation as corresponds to information contained in the first image.By one approach, this predicted occlusion information can be provided,at least in part, by processing the first image, at least in part, bydetermining occlusion value information for at least some pixels ascomprise the first image and then using the occlusion value informationfor the first image to determine a corresponding disparity informationmap for the first image.

By one approach, these teachings will also accommodate mapping contentsof the disparity information map for the first image to the second imageto thereby determine the aforementioned predicted occlusion information.This occlusion value information for the second image can then be usedto determine a corresponding disparity information map for the secondimage. By one approach, the resultant disparity information can then bemerged with the disparity information map for the first image mapped tothe second image, to provide a resultant disparity information map forthe second image.

These teachings will also accommodate, if desired, reworking theaforementioned first image disparity map in an iterative manner. Thiscan comprise, for example, mapping contents of the disparity informationmap for the second image to the first image to thereby determinesubsequent predicted occlusion information and re-computing a disparityvalue of each pixel in the first image by, at least in part, determininga likelihood of occlusion for at least some pixels comprising the firstimage using, at least in part, the subsequent predicted occlusioninformation. The subsequent predicted occlusion information can then beused to determine a corresponding subsequent disparity information mapfor the first image. These teachings will then accommodate mapping thecontents of the subsequent disparity information map for the first imageto the second image to thereby determine additional subsequent predictedocclusion information that can be used to determine a resultantdisparity information map for the second image.

Those skilled in the art will recognize and appreciate that theseteachings essentially provide for computing disparity and occludedregions together and then iterating the computation between the firstand second images. More particularly, by using a knowledge of whichpixels are occluded in the second image, a Markov Random Field-basedstereo algorithm can then serve to estimate more reliable disparityvalues for the pixels that are not occluded; the teachings set forthherein permit those occluded pixels to be predicted using previouscalculations. Such an approach tends to require less time and lessmemory than prior art approaches in this regard.

These and other benefits may become clearer upon making a thoroughreview and study of the following detailed description. Referring now tothe drawings, and in particular to FIG. 1, an illustrative process 100to facilitate processing a stereoscopic image using a first and a secondimage to facilitate computing a depth/disparity image that is compatiblewith many of these teachings will now be presented.

Following provision 101 of the aforementioned first and second images(which can generally otherwise accord with prior art practice in thisregard, if desired, and may comprise, for example, the right and leftimages as often correspond to stereoscopic images), this process 100 canoptionally provide for processing 102 the first image, at least in part,by determining occlusion value information for at least some pixels ascomprise that first image. This occlusion value information for thefirst image can then be used to determine a corresponding disparityinformation map for the first image. The contents of such a disparityinformation map for the first image can then be mapped to the secondimage to thereby determine corresponding predicted occlusioninformation.

Regardless of how obtained, this process 100 then provides for usingsuch predicted occlusion information as corresponds to informationcontained in the first image to compute 103 a disparity value of eachpixel in the second image by, at least in part, determining a likelihoodof occlusion for at least some pixels (and, if desired, all the pixels)as comprise the second image. This, of course, represents a significantpoint of departure as compares to numerous prior art approaches. Thepurpose, use, and significance of this step may become more evident inlight of the remaining description and provided details.

By one approach, this process 100 can then provide for merging 109 thedisparity information map for the first image with the disparityinformation map for the second image to provide a resultant disparityinformation map for the second image.

These teachings will also accommodate, however, the reworking of thefirst image disparity map prior to such a step. This can comprise, forexample, mapping 104 the contents of the disparity information map forthe second image to the first image to thereby determine subsequentpredicted occlusion information and then re-computing 105 a disparityvalue of each pixel in the first image by, at least in part, determininga likelihood of occlusion for at least some pixels comprising the firstimage using, at least in part, the aforementioned subsequent predictedocclusion information in an iterative fashion.

To illustrate, this subsequent predicted occlusion value information forthe first image can now be used 106 to determine a correspondingsubsequent disparity information map for the first image, followingwhich the contents of this subsequent disparity information map for thefirst image can be mapped 107 to the second image to thereby determineadditional subsequent predicted occlusion information. This additionalsubsequent occlusion information can then be used 108 to determine aresultant disparity information map for the second image.

Such reworking activities can, if desired, be repeated as many times asmay be desired. Eventually, however, and again as noted above, theresultant disparity information map for the first image can be merged109 with the disparity information map for the second image to provide aresultant disparity information map for the second image.

Further details regarding such steps are presented below. First,however, those skilled in the art will appreciate that theabove-described processes are readily enabled using any of a widevariety of available and/or readily configured platforms, includingpartially or wholly programmable platforms as are known in the art ordedicated purpose platforms as may be desired for some applications.Referring now to FIG. 2, an illustrative approach to such a platformwill now be provided.

As illustrated in FIG. 2, an apparatus 200 suitable to facilitate andembody these teachings through facilitation of the processing of astereoscopic image using a first and second image to facilitate, inturn, computing a depth/disparity image can be generally comprised of aprocessor 201 that operably couples to a memory 202. The memory 202serves to store the first and second image as well as any interveningdata sets as are contemplated by these teachings. Those skilled in theart will recognize that this memory 202 can be comprised of a singleintegral component or can be realized through a distributedarchitecture.

The processor 201 can comprise a fixed-purpose hardware platform or cancomprise a partially or wholly programmable platform. Such architecturalchoices are well known in the art and require no further elaborationhere. Such a processor 201 can be configured and arranged (via, forexample, corresponding programming as will be well understood by thoseskilled in the art) to carry out one of more of the steps andfunctionality describe herein. This can comprise at the least, forexample, providing the aforementioned first and second images andcomputing a disparity value of each pixel in the second image by, atleast in part, determining a likelihood of occlusion for at least somepixels as comprise the second image using, at least in part, predictedocclusion information as corresponds to information contained in thefirst image.

Those skilled in the art will recognize and understand that such anapparatus 200 may be comprised of a plurality of physically distinctelements as is suggested by the illustration shown in FIG. 2. It is alsopossible, however, to view this illustration as comprising a logicalview, in which case one or more of these elements can be enabled andrealized via a shared platform. It will also be understood that such ashared platform may comprise a wholly or at least partially programmableplatform as are known in the art.

Those skilled in the art will recognize and appreciate that theseteachings make use of a multi-phase occlusion handling approach forstereo imagery computations that is structurally simple and hence wellsuited to hardware-based implementations if desired. It will also berecognized that such an approach is generally more computationallyefficient than numerous existing approaches in this regard.Notwithstanding such benefits regarding simplicity and efficiency, theresults obtained via these teachings compare favorably with existingbest-in-class stereo computations. Furthermore, these benefits are wellsuited to support the use of such teachings in a video context,including a real time video context such as the real time rendering ofstreaming video content.

Referring now to FIG. 3, a somewhat more specific and detailedexplanation of a particular instantiation of these teachings will beprovided. Those skilled in the art will recognize and understand thatthis example is intended to serve only in an illustrative capacity andis not intended to comprise an exhaustive offering of all possibilitiesin this regard.

Following initialization, this process provides for per-pixel occlusioncost determinations to be made with respect to a Markov Random Field(MRF) stereo algorithm (using predicted occlusion information) 301 forthe right image of a stereoscopic reference. One then fills in 302 thedisparity map for this right image and then maps 303 the right disparityto the left image of the stereoscopic reference to thereby predict theoccluded region(s). Stereo MRF processing with per-pixel occlusion costdetermination 304 then occurs for the left image as a function, at leastin part, of the aforementioned predicted occluded region(s). Theresultant information is then used to fill in 305 a disparity map forthe left image.

To predict the occluded regions in the right (or left) image, one canstart by mapping each pixel to the right (or left) image according totheir disparity on the left (or right) image. The predicted occludedregions are those pixels in the right (or left) image that do not haveany pixels in the left (or right image) mapped to them.

At this point, a left disparity map A (as is determined as a function ofmapping 303 the right disparity to the left image to thereby predict theoccluded region(s)) can be merged 307 with a left disparity map B (as isdetermined upon filling in 305 the left disparity map) by taking, forexample, the MIN of the absolute disparity to yield a resultant leftdisparity map. If desired, following filling in 305 the left disparitymap, one can optionally map 306 the left disparity to the right image tothereby predict, again, the occluded region(s) and then re-enact thedescribed process using this presumably better starting point. Thisprocess can be so iterated any number of times as desired.

As noted, by one approach, these teachings can be employed with MarkovRandom Field (MRF) formulations as are already well known in the art.Presuming this context for the sake of example and not by way oflimitation, an optimal disparity map is the one that minimizes thefollowing MRF energy term:

$E = {{\sum{E_{Image}\left( {d(x)} \right)}} + {\sum\limits_{x_{1},{x_{2} \in N}}{E_{smooth}\left( {{d\left( x_{1} \right)},{d\left( x_{2} \right)}} \right)}}}$where E_(image) is the image matching cost, E_(smooth) is the smoothnesscost, and d(x) is the disparity of pixel x and N in a neighborhoodsystem (usually 4-connected where both “neighborhood systems” and“4-connectedness” are well understood concepts in the image processingand computer vision arts). E_(smooth) enforces the constraints thatneighboring pixels should have similar disparities. For the continuedsake of this example, and again not by way of limitation, examples ofthese two costs might include:E _(image)(d(x))=α min(|I _(L)(x)−I _(R)(x+d(x))|,γ₁)E _(smooth)(d(x ₁),d(x ₂))=β min(|d(x ₁)−d(x ₂)|,γ₂)*h(x ₁ ,x ₂)where alpha, beta, gamma_(—)1, and gamma_(—)2 are constants and h(x1,x2) represents a weighting (computed, for example, from the image).Alpha and beta are weighting that adjust the relative strength of theE_(smooth) term and the E_(image). They can be adjusted empirically tobalance the noise level and smoothness in the result disparity map.Gamma_(—)1 and gamma_(—)2 are two thresholds to limit the impact of anindividual pixel to the overall energy. Gamma_(—)1 can be determined bythe average absolute color differences between neighboring pixels.Gamma_(—)2 can be determined by the maximum disparity. H(x1, x2) can beany function that monotonically decreases based on the color differencesof pixel x1 and x2.

To handle occlusion, one can add an occlusion state that is equivalentto another disparity level having, however, these energy functions:E _(image)(d(x)=occ)=λ₁(x)E _(smooth)(d(x ₁)=occ,d(x ₂))=E _(smooth)(d(x ₁),d(x ₂)=occ)=λ₂ *h(x ₁,x ₂)E _(smooth)(d(x ₁)=occ,d(x ₂)=occ)=0where λ₁(x) is the per-pixel image cost for the occlusion state, λ₂ is aconstant, and occ means that the pixel is classified as occluded.Initially, λ₁(x) may also comprise a constant, but once the occlusionprediction becomes available as described above, this value can becalculated instead as:λ₁(x)=λ₀*[1−occ(x)]+λ_(occ) *occ(x)

Those skilled in the art will understand and recognize that thedisparity of occluded pixels cannot be likely computed from stereomatching. Instead, pursuant to these teachings, such information can befilled in by using the disparity of their neighboring non-occludedpixels on a same scanline as represented by the following expression:d(occ)=sign[d(x1)]*min(abs[d(x1)],abs[d(x2)]This fill in process can be efficiently implemented, if desired, byusing a two-pass scan of the entire image (for example, first from theleft side to the right side and then from the right side to the leftside). For at least some application settings it may be important tofirst perform the occlusion fill in step in the current disparity imagebefore mapping it to the other image to effect the occlusion predictionstep. Such a precaution, for example, may reduce the false alarm rate ofthe occlusion prediction activity.

So configured, those skilled in the art will recognize and appreciatethat these teachings can be implemented to achieve a per-pixel occlusioncost map that fits in a Markov Random Field formulation and that can befurther optimized, if desired, with a disparity computation at the sametime (using, for example, a belief propagation approach of choice). Theuse of an iterative approach that uses predicted occlusion informationfrom a most recently computed disparity map to generate an occlusioncost map provides a mechanism for achieving high quality results whilenevertheless greatly reducing immediate real time memory requirements.

Those skilled in the art will recognize that a wide variety ofmodifications, alterations, and combinations can be made with respect tothe above described embodiments without departing from the spirit andscope of the invention, and that such modifications, alterations, andcombinations are to be viewed as being within the ambit of the inventiveconcept. For example, those skilled in the art will recognize that theabove described modified energy function(s) can be further optimized, ina given application setting, by other methods applicable to MRF settingssuch as belief propagation.

1. A method to facilitate processing a stereoscopic image comprising afirst and second image to facilitate computing a depth/disparity image,comprising: computing, by a processor, a disparity map for the firstimage of the stereoscopic image using predicted occlusion informationfor the first image; computing, by the processor, a first disparity mapfor the second image of the stereoscopic image by mapping contents ofthe disparity map for the first image of the stereoscopic image to thesecond image of the stereoscopic image to determine predicted occlusioninformation for the second image of the stereoscopic image; determining,by the processor, a second disparity map for the second image of thestereoscopic image using the predicted occlusion information for thesecond image of the stereoscopic image; merging, by the processor, thefirst and second disparity maps for the second image of the stereoscopicimage to generate a resultant disparity map for just the second image ofthe stereoscopic image.
 2. The method of claim 1 further comprising:mapping contents of the second disparity map for the second image of thestereoscopic image to the first image of the stereoscopic image to,thereby, determine subsequent predicted occlusion information for thefirst image of the stereoscopic image.
 3. The method of claim 2 furthercomprising: re-computing the disparity map for the first image of thestereoscopic image using the subsequent predicted occlusion informationfor the first image of the stereoscopic image; and using the recomputeddisparity map for the first image of the stereoscopic image,re-iterating the steps of: mapping the contents of the disparity map forthe first image of the stereoscopic image to the second image of thestereoscopic image to determine the predicted occlusion information forthe second image of the stereoscopic image and to determine the firstdisparity map for the second image; determining the second disparity mapfor the second image; and merging the first and second disparity mapsfor the second image to generate the resultant disparity map for justthe second image.
 4. The method of claim 1, wherein the predictedocclusion information comprises likelihood of occlusion information. 5.The method of claim 4, wherein determining the second disparity map forthe second image using the predicted occlusion information for thesecond image comprises applying a Markov Random Field algorithm to thelikelihood of occlusion information.
 6. The method of claim 1, wherein:the first disparity map for the second image is determined directly frommapping the contents of the disparity map for the first image of thestereoscopic image to the second image; and the second disparity map forthe second image is determined by applying a Markov Random Fieldalgorithm to the predicted occlusion information for the second image.7. An apparatus to facilitate processing a stereoscopic image comprisinga first and second image to facilitate computing a depth/disparityimage, comprising: a memory having stored therein the first image of thestereoscopic image and the second image; a processor operably coupled tothe memory and being configured and arranged for: computing a disparitymap for the first image of the stereoscopic image using predictedocclusion information for the first image of the stereoscopic image;mapping contents of the disparity map for the first image of thestereoscopic image to the second image to determine predicted occlusioninformation for the second image and to determine a first disparity mapfor the second image; determining a second disparity map for the secondimage using the predicted occlusion information for the second image;merging the first and second disparity maps for the second image togenerate a resultant disparity map for just the second image.
 8. Theapparatus of claim 7, wherein the processor is further configured andarranged for: mapping contents of the second disparity map for thesecond image to the first image of the stereoscopic image to, thereby,determine subsequent predicted occlusion information for the first imageof the stereoscopic image.
 9. The apparatus of claim 7, wherein theprocessor is further configured and arranged for: re-computing thedisparity map for the first image of the stereoscopic image using thesubsequent predicted occlusion information for the first image of thestereoscopic image; and using the recomputed disparity map for the firstimage of the stereoscopic image, re-iterating the steps of: mapping thecontents of the disparity map for the first image of the stereoscopicimage to the second image to determine the predicted occlusioninformation for the second image and to determine the first disparitymap for the second image; determining the second disparity map for thesecond image; and merging the first and second disparity maps for thesecond image to generate the resultant disparity map for just the secondimage.
 10. The apparatus of claim 7, wherein the predicted occlusioninformation comprises likelihood of occlusion information.
 11. Theapparatus of claim 10, wherein the processor is further configured andarranged for: applying a Markov Random Field algorithm to the likelihoodof occlusion information to determine the second disparity map for thesecond image using the predicted occlusion information for the secondimage.
 12. The apparatus of claim 7, wherein the processor is furtherconfigured and arranged for: determining the first disparity map for thesecond image directly from mapping the contents of the disparity map forthe first image of the stereoscopic image to the second image; anddetermining the second disparity map for the second image applying aMarkov Random Field algorithm to the predicted occlusion information forthe second image.